Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20077/
• CommitID: 58be329
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20114/
• CommitID: 04552d1
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20120/
• CommitID: 04552d1
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20128/
• CommitID: 04552d1
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20150/
• CommitID: 3bf1fec
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20156/
• CommitID: 3bf1fec
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

@reta @nknize Will this lucene version change in 2.11 release? This is corresponding to an important decision-making in our feature. This is a re-invent feature and scheduled to release at 2.11.
For your information, we will introduce sparse semantic retrieval, which leverages lucene FeatureField and FeatureQuery. In older lucene version the FeatureQury performs bad because it cannot return real score upperbound to skip evaluating incompetitive docs. So we need to add a new settings to address the problem. However, in 9.8.0, lucene does many optimizations for Boolean query and we find we don't need this settings to boost the efficiency. This will affect our choice for field mapper and query clause.

@reta @nknize Will this lucene version change in 2.11 release? This is corresponding to an important decision-making in our feature. This is a re-invent feature and scheduled to release at 2.11.

@zhichao-aws if 9.8.0 happens to be released before 2.11, I think we could get it in, but I don't have a date for this release sadly [1]

[1] https://github.com/apache/lucene/milestones

@reta @nknize Will this lucene version change in 2.11 release? This is corresponding to an important decision-making in our feature. This is a re-invent feature and scheduled to release at 2.11.

@zhichao-aws if 9.8.0 happens to be released before 2.11, I think we could get it in, but I don't have a date for this release sadly [1]

[1] https://github.com/apache/lucene/milestones

I see, thanks!

@zhichao-aws For the record, @zhaih kicked off the 9.8 release discussion on the Lucene mailing list a couple of days ago: https://lists.apache.org/thread/xvhqdb0vzyfqsbpl9lvlrxxm2v4tr39z

@zhichao-aws For the record, @zhaih kicked off the 9.8 release discussion on the Lucene mailing list a couple of days ago: https://lists.apache.org/thread/xvhqdb0vzyfqsbpl9lvlrxxm2v4tr39z

That's good to know, thanks :)

Lucene has cut the 9.8 branch today https://github.com/apache/lucene/tree/branch_9_8. Can we make it to upgrade to lucene 9.8 in 2.x before 2.11 release?

Lucene has cut the 9.8 branch today https://github.com/apache/lucene/tree/branch_9_8. Can we make it to upgrade to lucene 9.8 in 2.x before 2.11 release?

Once the artifacts are published to Maven Central - for sure

Hey @reta We've got this weird little "hurry up" release locking next week (Oct 4th), and I'm worried we won't have a chance to absorb these changes with everything else going on. What would you think about holding merging this until after Oct 18th? Is there a reason we should to it sooner?

What would you think about holding merging this until after Oct 18th? Is there a reason we should to it sooner?

Hey @CEHENKLE , this was not backported to 2.x (we backport only releases), so non-issue :)

Ah coolio. Brainfail on my side. Thank you, @reta!

Hey, it looks like neural-search may be need these changes for 2.11 (I've asked them to comment on the issue). In the meantime, @reta (or others), do you know how impactful these changes would be to other parts of the core/the plugins if we merged them?

It's hard, because if we're going to do it, we need to go as soon as possible to give time for folks to absorb changes. But if we're not going to go, I don't want to backport and cause thrash. :(

(@nknize, @andrross for more eyes)

Technically this PR won't be the one to be backported, but rather #10276 (based on the released Lucene 9.8.0, which may be identical to the snapshot merged in this PR).

The differences likely to impact other parts of the code are probably the API changes: https://lucene.apache.org/core/9_8_0/changes/Changes.html#v9.8.0.api_changes, and changes to runtime behavior: https://lucene.apache.org/core/9_8_0/changes/Changes.html#v9.8.0.changes_in_runtime_behavior

Those changes to runtime behavior would potentially seem risky for our concurrent search work, but AFAIK, we're not seeing differences between 2.x and main -- right? (If anything, ensuring that 2.x and main behave the same there feels like that will reduce thrashing.)

Technically this PR won't be the one to be backported, but rather #10276 (based on the released Lucene 9.8.0, which may be identical to the snapshot merged in this PR).

@CEHENKLE as @msfroh pointed out, Apache Lucene 9.8.0 is released so technically we are good to go for 2.x (probably make it to 2.11). We don't see any regressions so far, the changes related to concurrent search seems to be properly tracked and accommodated.

Thanks @msfroh and @reta for pointing out. From neural-search side, Lucene 9.8 is needed so as not to overwirite the FeatureField implementation for performance issues and we don't want to deprecate in the future the parameter (for performance tuning) which we have to bring up in the condition of Lucene 9.7.

Meanwhile we are also aware of some potential risks of pushing #10276 forward:

1. Before OpenSearch 2.11, the main branch of ml-commons is behind 2.x on some commits. Although this has fixed at the moment, we still suspect that some other repos may have the same kind of problems. The main branch should catch up with 2.x with the latest features & implementations before we back-port Lucene 9.8 to 2.x, otherwise, there will be endless conflicts to deal with.
2. We believe that all repos have done their functional tests as well as some benchmarkings to their main branch with Lucene 9.8 snapshot and so far there was no negative feedback. But as a rigorous procedure, we suggest all the repo to reproduce their tests with the formal release artifacts of Luence 9.8 (not snapshot version) so as to validate whether there's some negative impact from the Lucene upgrade to OpenSearch. The neural search plugin is absorbing the performance improvement, we also want to see others' feedback.
3. We'd like to see whether the change of APIs or runtime behavior will have impact to AWS hosted search service which will subscribe this upgrade in the feature. That is another topic. We shall be aware but not discuss it on github.

Meanwhile we are also aware of some potential risks of pushing #10276 forward:

Thanks @model-collapse , with concerns you have raised it looks like we should not rush 9.8.0 into 2.11 branch (since the release is a few days away) but wait for cut off and plan for 2.12.

Before OpenSearch 2.11, the main branch of ml-commons is behind 2.x on some commits. Although this has fixed at the moment, we still suspect that some other repos may have the same kind of problems. The main branch should catch up with 2.x with the latest features & implementations before we back-port Lucene 9.8 to 2.x, otherwise, there will be endless conflicts to deal with.

My understanding is that ml-commons is unusual in terms of committing on 2.x directly. Other repos generally do the right thing: commit on main and backport to 2.x.

@model-collapse do you agree with below assessment from @reta? With the tight timelines for 2.11 release, ideally we don't want to force a Lucene upgrade.

Meanwhile we are also aware of some potential risks of pushing #10276 forward:

Thanks @model-collapse , with concerns you have raised it looks like we should not rush 9.8.0 into 2.11 branch (since the release is a few days away) but wait for cut off and plan for 2.12.

I agree with @msfroh that most repo will follow same pattern of commit to main and backport to 2.x in general. Therefore, I don't see a much risk on point 1 and point 2 especially we have been using lucene 9.8 snapshot in main branch for a while.

For point 3, is there any specific action item we generally take before lucene version bump? Shouldn't it be accessed and addressed during AWS hosted search service release in a separate timeline?

I agree with @msfroh that most repo will follow same pattern of commit to main and backport to 2.x in general. Therefore, I don't see a much risk on point 1 and point 2 especially we have been using lucene 9.8 snapshot in main branch for a while.

IMHO, I don't think we can equate testing of main to be equal to testing for 2.x. Just a simple diff (without any contextual/functional analysis) shows 5k+ files, 1400+ commits and 70k+ lines. Agree with assessment, lucene upgrade nearing 2.11 release window does feel risky.